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ABSTRACT 



Evaluations of the effectiveness of distance education 



courses may be influenced by differences between students who choose to take 
distance versus on-campus courses. At Utah State University, surveys were 
completed by 28 students who were enrolled in or had completed 
distance-education courses in the Special Education program and 39 students 
enrolled in similar classes on campus. The most notable differences between 
the groups were in age and family composition. Mean average age was 22 on 
campus and 43 off campus. One-quarter of campus students were married, but 
only 2 had children; 89 percent of distance students were married, and 75 
percent had children. Most students in both groups worked, but on- campus 
students were generally employed part-time. Over 85 percent of distance 
students worked full-time, and two -thirds worked in the public schools, 
primarily as teachers or paraprof essionals . Forty percent of the distance 
students and none of the campus students already had a bachelor's or master's 
degree. Almost all students were female and white. Implications for 
comparative evaluations are discussed. (SV) 
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COMPARATIVE EVALUATION OF DISTANCE AND ON CAMPUS EDUCATION: 

IS THE DECK STACKED? 

INTRODUCTION 

As more and more college content is provided via on-line and distance education courses, it is necessary to 
assess the efficacy of the courses delivered using these new media. Do distance learning courses prepare teachers 
adequately? Are teachers prepared as well as those taking on campus classes, or perhaps even better prepared? 
Before we can answer these questions we need to know the following: who is being served? and are the populations 
comparable? Various measures can be used to evaluate the teachers who emerge from each type of program. Yet, 
are these measures even valid if the populations are significantly different? The students are not randomly assigned, 
as one would do in a statistical study, but self-select their placement. What factors in this self-selection could effect 
a comparative evaluation? Is on-campus training preferable and does it produce teachers of a higher caliber? Are 
distance education students more highly motivated? 

While not claiming to answer each of the preceding questions, this study attempts to explore some of the 
possible differences between these populations. While limited to a survey of students in the Utah State University 
Department of Special Education and Rehabilitation, the results may suggest likely avenues for other researchers to 
explore before developing a comparative evaluation of distance education programs. 

METHODS 

Subjects 



Surveys were administered to students enrolled in the Utah State University Special Education 
undergraduate program during Fall Semester 2001, as well as last year’s distance education cohort. All distance 
education students were sent eMail requests to log onto a web-based survey page. In order to maximize returns, 
those students who had difficulty using the webpage or failed to complete the survey were sent a follow-up e-mail 
with the survey attached or imbedded. Twenty-eight distance education student responses were compiled. On 
campus, hard copies were printed and distributed to students enrolled in a comparable class. The printed surveys 
were identical in appearance to the web-based surveys. Thirty-nine surveys were returned and compiled from that 
group. 



These students represented the current cohort of on-campus and this and last year’s distance education 
populations. Since virtually all Special Education undergraduates take these classes either on campus or by distance 
education, it’s students would present a snapshot of the total enrollment. 

Setting 



The common setting for this study is the Special Education undergraduate program at Utah State 
University. Utah State is located in Logan, a small city in the northern part of the state which draws 90% of its 
students from the surrounding area of small cities and rural communities. This setting is split, however, by the 
manner of students’ course delivery medium. Some students are taking classes in a traditional campus class setting 
and the others are located well off campus around the state. The question, therefore, is not in what ways these 
settings differ, but how the student populations differ. 

Survey Content 

The questions on the survey were selected to highlight differences between the on campus and distant 
student populations. The researchers wanted to know if the populations differed in age and by how much; are there 
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differences in gender balance, marital status and number of children at home; and other pertinent factors. The 
questions were posed primarily in multiple choice format, such as “Are you ( ) married or ( ) single?” These 
questions were supplemented with a few very short answer queries: e.g. “What is your age?” Individual responses 
were transferred directly to a spreadsheet and either per cents or averages were computed. Little ambiguity or 
confusion resulted from the format, yet there was some of each. For example, students were asked for both their zip 
code and whether they lived in an urban, suburban, or rural setting. Approximately three-quarters of the on campus 
students listed their zip code as the zip code for Logan, a city of about 35,000 located 54 miles from the nearest 
larger city. None of these students indicated that they lived in an urban area, all but one choosing the suburban label 
instead. When asked how many hours they worked, if they were employed, several students indicated a range of 
hours, which is typical for part-time workers. The researchers chose to use the lower end of the range in coding the 
response, thus indicating a minimum number of hours worked weekly. 

Students were also asked for comments. Only eight wrote comments and they were not particularly 
germane to the study, thus they have been omitted. 

Survey Procedure 

With the intent of determining to what extent the populations of on-campus and distance education students 
were similar or different, the researchers discussed numerous questions that could have been included on a survey of 
this kind. Constraints on the length of a successful survey and the ability to compare results with other surveys 
narrowed both the number and the breadth of the questions. The remaining questions were formatted as a webpage. 
When completed by the respondents, the results were e-mailed automatically to the researcher. Distant students were 
requested by e-mail to complete the web survey which was directly linked from the e-mail itself Some students 
indicated problems with this technology, so a second survey was e-mailed directly to the class. Although 
respondents were assured of their anonymity, many gave their names on the survey. This was a fortunate choice, as 
some respondents attempted the web survey two or three times and unknowingly succeeded. Their extra responses 
were noted and not counted. 

For the on-campus class, the webpage survey was printed and handed out in class with very few changes 
(such as eliminating the line with a “button” to send the response). Students marked their responses and returned the 
surveys to the instructor who delivered them to the researcher. The completed surveys were numbered and then the 
individually identifying information severed from the rest of the form. Data was entered into a spreadsheet by the 
researcher and a student assistant, then the entries were recounted for an error check. Percentages or averages were 
calculated and are indicated in the tables. 

Results 



Seventy-six distance education students were requested to complete the survey: with 28 returns or 37%. 
On-campus 39 of 43 surveys were collected for a rate of 91%. Most of the requested information was complete, but 
there were a few responses with some areas left blank. This might have been due to response areas being formatted 
in two columns and the second column being overlooked. Where information was missing, calculations were made 
only on the information received. 

The most notable difference between the populations are in the areas of age and family composition. 
Average age on-campus was 22 years with a mode of 21(17 of 38 respondents) and a range from 20 to 5 1 . Off 
campus the mean was 43 with negligible mode (3 respondents were 31) and a range from 23 to 52. Twenty- four 
percent of the campus students were married, and only 2 of them had children, while 89% of the dist. ed. students 
were married, three-quarters of whom have children. 
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Table 1 





Age 


Married 


Children 


On-campus 


average: 22 yrs 


24% 


average: 0. 1 




mode: 21 yrs 




mode: 0 




range: 20-51 




range: 0-3 


Distance Ed. 


average: 43yrs 


89% 


average: 2 




mode: 3 1 yrs 




mode: 2 




range: 23-52 




range: 0-6 



More than three-quarters of students worked, but most on-campus students were employed part-time (16 
hours/week on average) and many distance ed. students were employed full-time (36 hours/week on average). 
Distance ed. students were primarily employed in the public schools: 32% as special education teachers, 18% as 
educational paraprofessionals, 10% as “other, education”, and 7% regular education teachers. 



Table 2 





Working 


HoursAVeek 


Primary Type of Job 


On-campus 


74% 


Average: 16 


59% not in education 


Distance Ed. 


86% 


Average: 36 


67% education 



The last area of marked difference between on-campus students and distance education students is current 
level of education. As one might expect from their youth, on-campus students have had less opportunity for higher 
degrees: Ninety-two percent have either a high school diploma (46%) or associate’s degree (46%). Much like Sieir 
diverse ages, distance education students are more varied in their previous education: 



Table 3 


21% High School 


32% Associate’s 


32% Bachelor’s 


7% Master’s 



Some questions revealed modest differences, with questionable significance. One hundred percent of the 
on-campus students and 86% of distance ed. students in this snapshot were female. Seventy-four percent on-campus 
and 86% off campus students were working while going to school. 

Several of the areas examined by the survey showed little if any significant differences between the two 
student populations. Eleven percent of distance education students said they had a disability to the 8% on-campus. 
Reflecting the lack of ethnic diversity in Utah, 93% of distance education and 100% of on-campus students reported 
being Caucasian. The lone non-white student only claimed to be “other.” Though there was considerable difference 
in highest degree obtained, grade point average for each group was virtually the same (3.43 dist. ed. vs. 3.46 on- 
campus). 

DISCUSSION 

Conclusions 



As the technology used for distance education as evolved, so has the technology to evaluate it. Of the 
several ways to assess distance learning, many compare data against that of traditional on-campus courses. In 
several aspects, we found that the student populations in each educational setting are similar, yet there are some 
distinct and serious dissimilarities with profound implications for both instruction and evaluation. 
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Implications 



Holding populations the same, as the following researchers have tried to do, validates comparison. The 
differences in student populations that we found shows a crucial need for future researchers to be wary of the 
problems in making comparison studies. In 1995 Bruce Thyer used an attitudinal survey to compare his on-campus 
and distance education classes and found no difference. However, are attitudinal surveys - course satisfaction for 
example - comparable when one population may consists of mostly 21 year olds, and ^e other a spectrum averaging 
twice as old? Thyer recommended “well crafted comparison studies that examine not simply student attitudes ..., but 
actual knowledge and skills.” (Thyer, 1995) 

In an attempt to look at actual learning, evaluators of the Oklahoma Televised Instruction System compared 
students’ grades. These researchers found that contrary to the prevailing wisdom at the time, “distance students are 
not adversely affected relative to course performance... In fact the reverse may be true.” (Dillon & Gunawardena, 
1995) Grades may be a useful cross discipline measure, but one must be cautious as grades are somewhat subjective. 
Instructors often make accommodations, and do we know if groups are, in fact, treated similarly. Should one use the 
same outcome measures with mostly high school graduates as with a mix of high school, associate’s, bachelor’s and 
master’s degree holders? 

In 1999, Rodriques evaluated for quality of content face-to-face discussions with participants in on-online 
forums. She found that online, “student’s comments were richer, fuller and more reflective because they had 
opportunities to deliberate, phrase and actively construct their responses.” (Rodriques, 1999) Could comparing the 
depth and quality of discussion among older students who are working and raising families to that of single young 
adults successfully yield information about the instructional medium? 

This study we undertook was limited to one department of one college at one university. However, the 
question of population differences applies to any distance education comparison, as well as to course development, 
improvement, and selection. 
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